Learning to combine the modalities of language and video for temporal moment localization

Temporal moment localization aims to retrieve the best video segment matching a specified by query. The existing methods generate visual and semantic embeddings independently fuse them without full consideration of long-term temporal relationship between them. To address these shortcomings, we introduce novel recurrent unit, cross-modal long short-term memory (CM-LSTM), mimicking human cognitive process localizing moments that focuses on part related query, accumulates contextual information across entire recurrently. In addition, devise two-stream attention mechanism for both attended unattended features input query prevent necessary from being neglected. obtain more precise boundaries, propose attentive interaction network (TACI) generates two 2D proposal maps obtained globally integrated features, which are generated using CM-LSTM, locally boundary score sequences then combines into final map in an end-to-end manner. On TML benchmark dataset, ActivityNet-Captions, TACI outperforms state-of-the-art with [email protected] 45.50% 27.23% protected], respectively. show revised state-of-the-arts replacing original LSTM our CM-LSTM achieves performance gains.